{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Objective\n",
    "\n",
    "We will be implementing the Single Shot Multibox Detector (SSD), a popular, powerful, and especially nimble network for this task. The authors' original implementation can be found here.\n",
    "\n",
    "Here are some examples of object detection in images not seen during training –\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/blob/master/img/baseball.gif?raw=true)\n",
    "\n",
    "# Concepts\n",
    "- Object Detection 目标检测\n",
    "\n",
    "- Single-Shot Detection 端到端的目标检测算法，早期的目标检测算法有两个步骤stages: region proposal network 首先提出一些可能的区域，再利用一个分类器对这些可能的区域进行识别。 计算上非常昂贵，不适合于现实世界中的实时检测应用。 而single-stage的方法将兴趣区域定位和检测任务都封装在单个前向sweep中，大大加快了目标检测的速度。\n",
    "\n",
    "- Multiscale Feature Maps. 多尺度特征图 在图像分类任务中，我们的预测基于最终的卷积特征图-原始图像的最小但最深的表示形式。（利用最后一层提取到的高层抽象特征），在目标检测中，来自中间卷积层的特征图也可以直接有用，因为它们以不同的比例表示原始图像。因此，在不同特征图上运行的固定大小的过滤器将能够检测各种大小的对象。\n",
    "\n",
    "- Priors 先验，一些预先设置好的框，它们被定义在特定特征图上的特定位置，具有特定的宽高比和尺度。 这些先验知识被仔细地选择，以符合数据集中目标对象边界框地特征。（Ground Truths）\n",
    "\n",
    "- Multibox 这是一种将预测对象的边界框表示为**回归问题**的技术，其中，将检测到的对象的坐标回归到其地面实况的坐标。另外，对于每个预测的盒子，针对各种对象类型生成分数。先验可作为预测的可行起点，因为它们是根据基本事实建模的，Therefore, there will be as many predicted boxes as there are priors, most of whom will contain no object.\n",
    "\n",
    "- Hard Negative Mining. 这是指明确选择模型预测的最严重错误肯定，并迫使其从这些示例中学习。换句话说，我们仅挖掘那些最难正确识别模型的负例。在对象检测的情况下，绝大多数预测框都不包含对象，这也有助于减少负-正不平衡\n",
    "\n",
    "- Non-Maximum Suppression 非极大值抑制 在任何给定位置，多个先验可能会明显重叠。因此，由这些先验产生的预测实际上可能是同一对象的重复。非最大抑制（NMS）是一种通过抑制除得分最高的预测之外的所有预测来消除冗余预测的方法\n",
    "\n",
    "# Overview\n",
    "\n",
    "## Boundary coordinates\n",
    "\n",
    "一个边界框可以简单表示为:$(x_min,y_min,x_max, y_max)$\n",
    "\n",
    "同时$x_min$等可以进行归一化，变成与图像大小无关的0-1的相对数据。\n",
    "\n",
    "Center-Size coordinates 也可以表示一个边界框，这个是更合理的表达方式，利用$(c_x,c_y,w,h)$来表示一个边界框。\n",
    "\n",
    "Jaccard Index：交并比，基本概念：\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/blob/master/img/jaccard.jpg?raw=true)\n",
    "这里说几点不那么被关注的知识点：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single Shot Detector (SSD)\n",
    "\n",
    "SSD 是一个卷积神经网络，可以被分为三个部分\n",
    "- Base convolutions derived from an existing image classification architecture that will provide lower-level feature maps.\n",
    "\n",
    "- Auxiliary convolutions added on top of the base network that will provide higher-level feature maps.\n",
    "\n",
    "- Prediction convolutions that will locate and identify objects in these feature maps.\n",
    "\n",
    "基础部分，提取低层次特征，辅助部分，提取高层次特征，预测部分，在特征图中定位识别目标。\n",
    "\n",
    "SSD300和SSD512 是输入图像大小不一样，512高分辨率更复杂一点，效果更好。但是基本原理类似，所以以SSD300为例。\n",
    "\n",
    "### Base Convolutions – part 1 基础卷积部分\n",
    "\n",
    "从图像分类出发，CNN在图像分类中取得了分层好的效果，而且这些模型通过大量的实验被证明在图像分类中可以非常好地区分不同地图像，说明这些模型能够很好地捕获图像的本质特征，基本特性，basic essence；\n",
    "\n",
    "而且通过在ImageNet上预训练的分类网络，可以通过迁移学习很好地获取这种特征提取的能力，所以通过从不同但密切相关的任务中借用知识，我们甚至在开始之前就已经取得了进步。 \n",
    "\n",
    "作者利用VGG-16体系结构作为其基础网络。\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/vgg16.PNG)\n",
    "\n",
    "> They recommend using one that's pretrained on the ImageNet Large Scale Visual Recognition Competition (ILSVRC) classification task. 作者建议用imagenet预训练网络作为基础网络，这些预训练网络在pytorch中都可以找到，不同的网络结构都有，按实际要求配置。\n",
    "\n",
    "根据本文，我们必须对该预训练网络进行一些更改，以使其适应我们自己的目标检测挑战。有些是合乎逻辑且必要的，而另一些则主要是出于方便或偏爱\n",
    "\n",
    "* The input image size will be 300, 300, as stated earlier.\n",
    "\n",
    "* The 3rd pooling layer, which halves dimensions, will use the mathematical ceiling function instead of the default floor function in determining output size. This is significant only if the dimensions of the preceding feature map are odd and not even. By looking at the image above, you could calculate that for our input image size of 300, 300, the conv3_3 feature map will be of cross-section 75, 75, which is halved to 38, 38 instead of an inconvenient 37, 37.\n",
    "\n",
    "*  We modify the 5th pooling layer from a 2, 2 kernel and 2 stride to a 3, 3 kernel and 1 stride. The effect this has is it no longer halves the dimensions of the feature map from the preceding convolutional layer.\n",
    "\n",
    "* We don't need the fully connected (i.e. classification) layers because they serve no purpose here. We will toss fc8 away completely, but choose to rework fc6 and fc7 into convolutional layers conv6 and conv7.\n",
    "\n",
    "The first three modifications are straightforward enough, but that last one probably needs some explaining."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FC → Convolutional Layer\n",
    "How do we reparameterize a fully connected layer into a convolutional layer?\n",
    "全连接层是要将特征图展开，然后进行线性运算，如图\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/fcconv1.jpg)\n",
    "\n",
    "如果将全连接层替换为一个卷积层，输出同时也是与上述全连接层一致，那么需要如下图的卷积核。\n",
    "现在，考虑一个不同的场景，其中我们使用卷积层来产生2个输出值。\n",
    "\n",
    "在这里，显然不必将尺寸2、2、3的图像弄平。 卷积层使用两个具有与图像相同形状的12个元素的滤镜来执行两个点积。 这两个以灰色显示的滤波器是卷积层的参数。\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/fcconv2.jpg)\n",
    "\n",
    "但这是关键部分–在两种情况下，输出Y_0和Y_1都是相同的！\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/fcconv3.jpg)\n",
    "这告诉我们什么？\n",
    "\n",
    "在具有I个输入通道的尺寸为H，W的图像上，输出尺寸为N的完全连接层等效于内核尺寸等于图像尺寸H，W和N的输出通道的卷积层，但前提是 全连接网络N，H * W * I与卷积层N，H，W，I的参数相同。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/fcconv4.jpg)\n",
    "\n",
    "因此，只需将其参数reshape , 即可将任何**全连接的层转换为等效的卷积层**。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Base Convolutions – part 2\n",
    "We now know how to convert fc6 and fc7 in the original VGG-16 architecture into conv6 and conv7 respectively.\n",
    "\n",
    "In the ImageNet VGG-16 shown previously, which operates on images of size 224, 224, 3, you can see that the output of conv5_3 will be of size 7, 7, 512. Therefore –\n",
    "\n",
    "fc6 with a flattened input size of 7 * 7 * 512 and an output size of 4096 has parameters of dimensions 4096, 7 * 7 * 512. The equivalent convolutional layer conv6 has a 7, 7 kernel size and 4096 output channels, with reshaped parameters of dimensions 4096, 7, 7, 512.\n",
    "\n",
    "fc7 with an input size of 4096 (i.e. the output size of fc6) and an output size 4096 has parameters of dimensions 4096, 4096. The input could be considered as a 1, 1 image with 4096 input channels. The equivalent convolutional layer conv7 has a 1, 1 kernel size and 4096 output channels, with reshaped parameters of dimensions 4096, 1, 1, 4096.\n",
    "\n",
    "We can see that conv6 has 4096 filters, each with dimensions 7, 7, 512, and conv7 has 4096 filters, each with dimensions 1, 1, 4096.\n",
    "\n",
    "These filters are numerous and large – and computationally expensive.\n",
    "\n",
    "To remedy this, the authors opt to reduce both their number and the size of each filter by subsampling parameters from the converted convolutional layers.\n",
    "\n",
    "conv6 will use 1024 filters, each with dimensions 3, 3, 512. Therefore, the parameters are subsampled from 4096, 7, 7, 512 to 1024, 3, 3, 512.\n",
    "\n",
    "conv6 will use 1024 filters, each with dimensions 1, 1, 1024. Therefore, the parameters are subsampled from 4096, 1, 1, 4096 to 1024, 1, 1, 1024.\n",
    "\n",
    "Based on the references in the paper, we will subsample by picking every mth parameter along a particular dimension, in a process known as decimation.\n",
    "\n",
    "Since the kernel of conv6 is decimated from 7, 7 to 3, 3 by keeping only every 3rd value, there are now holes in the kernel. Therefore, we would need to make the kernel dilated or atrous.\n",
    "\n",
    "This corresponds to a dilation of 3 (same as the decimation factor m = 3). However, the authors actually use a dilation of 6, possibly because the 5th pooling layer no longer halves the dimensions of the preceding feature map.\n",
    "\n",
    "We are now in a position to present our base network, the modified VGG-16.\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/modifiedvgg.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above figure, pay special attention to the outputs of conv4_3 and conv_7. You will see why soon enough.\n",
    "\n",
    "上面说的主要是，原始的特征图计算方式，计算量太大，可以缩小一点特征图的大小，减少计算量，所以原始的网络可以被转换为如上图所示的一个全卷积网络进行特征提取。在提取完特征后，通过一个辅助卷积网络，进行多尺度的特征提取。\n",
    "\n",
    "### Auxiliary Convolutions\n",
    "现在，我们将在基础网络之上堆叠一些卷积层。这些卷积提供了附加的特征图，每个特征图都逐渐小于最后一个。\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/auxconv.jpg)\n",
    "\n",
    "我们引入了四个卷积块，每个都有两层。同时通过在基础网络中进行池合并来减小大小，但在此，第二层的步幅为2有助于实现。\n",
    "\n",
    "## 预测， 我们是预测什么呢？\n",
    "\n",
    "在继续进行预测卷积之前，我们必须首先了解我们正在预测什么。当然，它是物体及其位置，但是以什么形式？在这里，我们必须了解先验知识及其在SSD中的关键作用。\n",
    "\n",
    "### 先验\n",
    "\n",
    "如果没有先验，对象所在的位置可以是任意位置，大小，形状，这样在整个图像上，可能会有无穷个可能（不准确，大概意思）但是很多的可能，是根本不可能的，所以对于这些位置，我们要做一个预估，利用先验知识过滤掉一些不可能的情况，实际上，我们可以将潜在预测的数学空间离散为数千种可能性。\n",
    "\n",
    "先验是预先计算的，固定的盒子，它们共同代表了可能的盒子和近似盒子的预测范围。\n",
    "\n",
    "先验是根据我们数据集中地面真实对象的形状和大小手动选择的，但要谨慎选择。通过将这些先验放置在特征图中的每个可能位置，我们还考虑了位置的变化。\n",
    "\n",
    "在定义先验时，作者指出-\n",
    "\n",
    "它们将应用于各种低级和高级功能图，即。那些conv4_3，conv7，conv8_2，conv9_2，conv10_2，和conv11_2。这些是前面图中指示的相同特征图。\n",
    "\n",
    "如果先验有比例尺s，那么它的面积等于有边的正方形的面积s。最大的特征图conv4_3将具有先验，其比例尺为0.1，即10%图像的尺寸，而其余要素的先验先后比例尺从0.2到线性增加0.9。如您所见，较大的特征图具有较小的先验比例，因此非常适合检测较小的对象。\n",
    "\n",
    "在特征图上的每个位置，都会有各种纵横比的先验。所有功能图都具有先验的比率1:1, 2:1, 1:2。的中间特征的地图conv7，conv8_2以及conv9_2将还具有比先验3:1, 1:3。而且，所有特征图将具有一个额外的优先级，其纵横比为，1:1且其比例为当前特征图和后续特征图的比例尺的几何平均值。\n",
    "\n",
    "| Feature Map From | Feature Map Dimensions | Prior Scale | Aspect Ratios | Number of Priors per Position | Total Number of Priors on this Feature Map |\n",
    "| :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n",
    "| `conv4_3`      | 38, 38       | 0.1 | 1:1, 2:1, 1:2 + an extra prior | 4 | 5776 |\n",
    "| `conv7`      | 19, 19       | 0.2 | 1:1, 2:1, 1:2, 3:1, 1:3 + an extra prior | 6 | 2166 |\n",
    "| `conv8_2`      | 10, 10       | 0.375 | 1:1, 2:1, 1:2, 3:1, 1:3 + an extra prior | 6 | 600 |\n",
    "| `conv9_2`      | 5, 5       | 0.55 | 1:1, 2:1, 1:2, 3:1, 1:3 + an extra prior | 6 | 150 |\n",
    "| `conv10_2`      | 3,  3       | 0.725 | 1:1, 2:1, 1:2 + an extra prior | 4 | 36 |\n",
    "| `conv11_2`      | 1, 1       | 0.9 | 1:1, 2:1, 1:2 + an extra prior | 4 | 4 |\n",
    "| **Grand Total**      |    –    | – | – | – | **8732 priors** |\n",
    "\n",
    "\n",
    "特征图大小为38*38, 每个像素有4个框，那样就有 $38\\times38\\times4 = 5776$个锚框，所以这样加起来\n",
    "\n",
    "特征图一共会生成8732个锚框。 这里也叫先验\n",
    "\n",
    "There are a total of 8732 priors defined for the SSD300!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 可视化锚框 先验\n",
    "这里定义 尺度 scales 和 宽高比 aspect ratios\n",
    "\n",
    "$$\n",
    "w \\times h = s\n",
    "$$\n",
    "$$\n",
    "\\frac{w}{h} = a\n",
    "$$\n",
    "\n",
    "一般来说，定义锚框是定义s和a, 通过上述关系，可以将w,h求出。\n",
    "\n",
    "$$\n",
    "w = s \\times \\sqrt{a}\n",
    "$$\n",
    "\n",
    "$$\n",
    "h = \\frac{s} {\\sqrt{a}}\n",
    "$$\n",
    "例如，让我们尝试在conv9_2中的特征图的中心图块上显示先验的样子\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/priors1.jpg)\n",
    "\n",
    "\n",
    "其他位置锚框的位置示意图\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/priors2.jpg)\n",
    "\n",
    "### Predictions vis-à-vis Priors 先验 与 预测\n",
    "我们说过，我们将使用回归来找到对象边界框的坐标。但是，当然，先验不能代表我们的最终预测值吗？\n",
    "\n",
    "再次重申，先验**大致代表了预测的可能性**。\n",
    "\n",
    "这意味着我们使用**每个先验作为近似起点**，然后**找出需要调整多少**以**获取对边界框**的**更精确的预测**。\n",
    "\n",
    "因此，如果**每个预测的边界框**与**先验框都略有偏差**，而我们的目标是**计算该偏差**，则需要一种方法来对其进行度量或量化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测偏移量， 而不是准确的坐标 \n",
    "考虑一只猫，它的预测边界框以及进行预测的先验条件。\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/ecs1.PNG)\n",
    "\n",
    "假设它们以我们熟悉的中心尺寸坐标表示。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/ecs2.PNG)\n",
    "\n",
    "考虑到对每个先验进行了调整以获得更精确的预测，这四个偏移量（g_c_x，g_c_y，g_w，g_h）是回归边界框坐标的形式。\n",
    "\n",
    "如您所见，每个偏移量都通过先验的相应尺寸进行归一化。这是有道理的，因为某个偏移量对于较大的先验比对于较小的先验将不那么重要。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测卷积\n",
    "之前，我们为六个不同比例和粒度的特征图指定了先验并为其定义了先验。那些conv4_3，conv7，conv8_2，conv9_2，conv10_2，和conv11_2。\n",
    "然后，对于每个预先在每个上的位置各特征图，我们要预测-\n",
    "\n",
    "* 边界框的偏移量(g_c_x, g_c_y, g_w, g_h)。\n",
    "\n",
    "* 边界框的一组n_classes分数，其中n_classes代表对象类型（包括背景类）的总数。\n",
    "\n",
    "为了以最简单的方式做到这一点，我们为每个特征图都需要**两个卷积层** –\n",
    "\n",
    "- 一个局部化预测卷积层 localization prediction  ，3, 3大小， 在每个位置使用核进行评估（with padding and stride of 1），并为该位置上存在的每个先前对象使用4个过滤器。\n",
    "\n",
    "- 4个先验的滤波器计算(g_c_x, g_c_y, g_w, g_h)从该先验预测的边界框的四个编码偏移。\n",
    "\n",
    "- 一个类预测卷积层，其3, 3内核在每个位置（即的填充和跨度1）进行评估，并使用该位置的每个先验n_classes过滤器。\n",
    "\n",
    "n_classes先验的过滤器计算该先验的一组n_classes分数。\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/predconv1.jpg)\n",
    "\n",
    "All our filters are applied with a kernel size of 3, 3.我们实际上并不需要与先验形状相同的内核（或过滤器），因为不同的过滤器将学习对不同的先验形状做出预测。等着看这些卷积的输出。再次考虑conv9_2中的特征图。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/predconv2.jpg)\n",
    "\n",
    "定位层和类别预测层的输出分别以蓝色和黄色显示。您可以看到横截面（5，5）保持不变。我们真正感兴趣的是三维，即通道。这些包含实际的预测。 如果您在本地化预测中选择一个图块（任何图块）并将其展开，您会看到什么？\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/predconv3.jpg)\n",
    "\n",
    "24个通道里，包含了6个bbox的偏移量。定位预测的每个位置处的信道值表示相对于该位置处的先验的编码偏移。\n",
    "\n",
    "现在，对类预测进行相同的操作。假设n_classes = 3。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/predconv4.jpg)\n",
    "\n",
    "与以前类似，这些渠道代表该位置上先验的班级得分。现在我们了解了conv9_2对特征图的预测是什么样的，我们可以将其重塑为更合适的形式。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/reshaping1.jpg)\n",
    "\n",
    "我们已经连续安排了150个预测。对人类而言，这应该看起来更直观。 但是，我们不要在这里停下来。我们可以对所有图层的预测执行相同的操作，然后将它们堆叠在一起。\n",
    "\n",
    "我们之前计算得出，我们的模型总共定义了8732个先验。因此，将有8732个编码偏移形式的预测框和8732组类分数。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/reshaping2.jpg)\n",
    "\n",
    "这是预测阶段的最终输出。一堆盒子，如果愿意的话，并估算其中的内容。\n",
    "\n",
    "但是8000个框，也略多了一点，怎么会有那么多框呢，所以，要进行一定的筛选。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multibox loss\n",
    "\n",
    "根据我们的预测性质，很容易看出为什么我们可能需要唯一的损失函数。我们中的许多人以前曾在回归或分类设置中计算过损失，但很少（如果有的话）**在一起**。 多个损失合在一起。\n",
    "\n",
    "显然，我们的总损失必须是两种类型的预测（边界框本地化和类分数）的总和。\n",
    "\n",
    "有一些问题需要回答\n",
    "- 回归框将使用什么损失函数？\n",
    "\n",
    "- 我们是否将多类交叉熵用于类分数？\n",
    "\n",
    "- 我们将以什么比例合并它们？\n",
    "\n",
    "- 我们如何将预测的盒子与他们的事实相匹配？\n",
    "\n",
    "- 我们有8732个预测！这些中的大多数都不会包含任何对象吗？我们甚至考虑他们吗？\n",
    "\n",
    "### 将预测与真实情况相匹配\n",
    "\n",
    "请记住，任何**有监督的学习算法的难点**在于，我们需要能够将预测与他们的基本事实ground truth相匹配。这很棘手，因为对象检测比普通学习任务更开放。\n",
    "\n",
    "为了使模型学习任何东西，我们需要以允许我们的预测与图像中实际存在的对象之间进行比较的方式来构造问题。\n",
    "\n",
    "先验可以告诉我们这些问题：\n",
    "\n",
    "* 找到所有先验与ground truth的交并比  Find the Jaccard overlaps between the 8732 priors and N ground truth objects. This will be a tensor of size 8732, N.\n",
    "\n",
    "* 将8732的每个先验匹配到与其重叠最大的对象。作为class\n",
    "\n",
    "* 小于一定交并比的对象，我们认为是负例，背景类。  如果先验条件与Jaccard重叠小于0.5的对象匹配，则不能说它“包含”该对象，因此是否定匹配。考虑到我们有成千上万个先验，大多数先验将测试一个对象是否定的。 \n",
    "\n",
    "* 另一方面，少数先验实际上会与一个对象显着重叠（大于0.5），并且可以说是“包含”该对象。这些是正面比赛。\n",
    "\n",
    "* 既然我们已经将8732的每个先验与基本事实进行了匹配，实际上，我们也将相应的8732预测与基本事实进行了匹配。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/matching1.PNG)\n",
    "\n",
    "为了方便起见，我们假设只有七个先验，以红色显示。地面真相为黄色-该图像中包含三个实际对象。按照前面概述的步骤进行操作将产生以下匹配结果–\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/matching2.jpg)\n",
    "\n",
    "现在，每个先验都有一个正或负的匹配项。通过扩展，每个预测都有一个正或负的匹配项。\n",
    "现在，与某个对象正面匹配的预测具有地面真实坐标，这些坐标将作为定位的目标，即在回归任务中。自然，没有否定匹配的目标坐标.\n",
    "\n",
    "所有预测都具有GT 标签，如果是正向匹配，则为对象的类型；如果是负向匹配，则为背景类。这些用作类别预测的目标，即分类任务。\n",
    "\n",
    "## Localization loss 定位损失\n",
    "\n",
    "背景类，不进行损失计算，只有有对象的类，才进行损失计算。\n",
    "\n",
    "对于否定匹配，我们没有地面真相坐标。这是很合理的。为什么训练模型在空白空间周围绘制盒子？\n",
    "\n",
    "由于我们以偏移量的形式预测了定位框，(g_c_x, g_c_y, g_w, g_h)因此在计算损失之前，还需要对地面真实坐标进行相应的编码。\n",
    "\n",
    "Localization loss 是在正匹配的 定位框的编码 **偏移量** 与它们的Ground Truth之间的平均平滑L1损失。\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/locloss.jpg)\n",
    "\n",
    "## Confidence loss 置信度损失\n",
    "\n",
    "每个预测，无论是正面还是负面的，都具有与之相关的地面真相标签。该模型识别两个对象以及它们的缺乏很重要。\n",
    "\n",
    "但是，考虑到图像中通常**只有少数几个对象**，因此我们做出的数千个预测中的**绝大多数实际上并不包含对象**。就像沃尔特·怀特（Walter White）所说的那样，轻踩一下。\n",
    "\n",
    "如果**否定匹配压倒了肯定匹配**(类别不平衡问题，如果全部丢进去训练，肯定出现偏向对负例的预测)，我们将得到一个模型，该模型**不太可能检测到对象，因为通常会教它检测背景类**。\n",
    "\n",
    "解决方案可能很明显–**限制将在损失函数中评估的否定匹配数**。但是我们该如何选择？\n",
    "\n",
    "好吧，为什么不使用模型最错误的模型呢？换句话说，仅在模型发现最难识别没有对象的地方使用那些预测。这被称为Hard Negative Mining.。\n",
    "\n",
    "###  正负例不平衡问题\n",
    "例如，我们将使用的hard negatives 数量N_hn通常是此图像的positive matches数量的固定倍数。在这种特殊情况下，作者决定使用三倍的hard negatives ，即N_hn = 3 * N_p。通过为每个负匹配的预测找到交叉熵损失并选择N_hn损失最大的预测，可以发现最难的hard matches 。\n",
    "\n",
    "那么，置信度损失就是正负匹配中交叉熵损失的总和。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/confloss.jpg)\n",
    "\n",
    "您会注意到，它是通过正面匹配的数量得出的平均值。\n",
    "\n",
    "## Total loss\n",
    "\n",
    "The Multibox loss is the aggregate of the two losses, combined in a ratio α.\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/totalloss.jpg)\n",
    "\n",
    "通常，我们不必决定α的值。它可能是一个可学习的参数。 但是，对于SSD，作者只需使用α= 1，即将两个损耗相加。我们接受！\n",
    "\n",
    "## Processing predictions\n",
    "\n",
    "训练模型后，我们可以将其应用于图像。但是，这些**预测仍然是原始形式**-**两个张量包含8732个先验的偏移量和类分数**。这些将需要进行处理以获得带有标签的最终的，可人类解释的边界框。\n",
    "\n",
    "这需要以下内容–\n",
    "\n",
    "我们有8732个预测框，表示为相对(g_c_x, g_c_y, g_w, g_h)于其各自先验的偏移量。将它们解码为实际上可以直接解释的边界坐标。\n",
    "\n",
    "- 然后，对于每个非背景类，\n",
    "\n",
    "- 为每个8732个框提取该类的分数。\n",
    "\n",
    "- 消除不符合此分数特定阈值的框。\n",
    "\n",
    "- 其余（未排除）框是此特定对象类别的候选项。\n",
    "\n",
    "此时，如果您要在原始图像上绘制这些候选框，则会看到许多高度重叠的框，这些框显然是多余的。这是因为在我们可以使用的数千个先验条件中，很可能有多个预测对应于同一对象。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/nms1.PNG)\n",
    "\n",
    "显然只有三个物体–两只狗和一只猫。但是，根据该模型，有3种狗和2只猫。\n",
    "\n",
    "请注意，这只是一个温和的例子。确实可能更糟。\n",
    "\n",
    "现在，对于您来说，很明显哪些框在引用同一对象。这是因为您的大脑可以处理某些盒子和一个特定对象之间的重合。\n",
    "\n",
    "在实践中，将如何进行？\n",
    "\n",
    "首先，根据他们的可能性排列每个班级的候选人。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/nms2.PNG)\n",
    "\n",
    "我们已经按照他们的分数对其进行了排序。\n",
    "\n",
    "下一步是找到**哪些候选者是多余的**。我们已经可以使用一种工具来判断两个框之间的共同点-Jaccard重叠, 通过交并比来判断。\n",
    "\n",
    "因此，如果我们要绘制给定类别中所有候选项之间的**Jaccard相似性**，我们可以评估每对货币对，如果发现它们之间存在**显着重叠**，则**仅保留更可能的候选项**。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/nms3.jpg)\n",
    "因此，我们淘汰了流氓候选人-每只动物之一。\n",
    "\n",
    "此过程称为非最大抑制（NMS），因为当发现多个候选者彼此之间有明显重叠从而可以引用相同的对象时，我们将压制除得分最高的对象以外的所有对象。\n",
    "\n",
    "从算法上讲，它的执行过程如下：\n",
    "\n",
    "- 为每个非背景类别选择候选对象后，\n",
    "\n",
    "- 按照降低可能性的顺序安排该类别的候选人。\n",
    "\n",
    "- 考虑得分最高的候选人。消除所有分数较低且Jaccard重叠大于0.5该候选人的候选人。\n",
    "\n",
    "- 考虑仍在池中的下一个得分最高的候选人。消除所有分数较低且Jaccard重叠大0.5于此候选人的候选人。\n",
    "\n",
    "重复直到遍历整个候选序列。\n",
    "\n",
    "最终结果是，对于图像中的每个对象，您只有一个框-最好的一个框。\n",
    "\n",
    "![](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection/raw/master/img/nms4.PNG)\n",
    "\n",
    "非最大抑制对于获得质量检测至关重要。\n",
    "\n",
    "幸运的是，这也是最后一步。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "684.667px",
    "left": "706px",
    "top": "109.333px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
